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Do  not  assume  content  reflects  current 
scientific  knowledge,  policies,  or  practices. 

- 


This  is  an  introduction  to  evaluation  of  Extension  educational  work.  It  includes  some 
basic  principles  of  evaluation,  and  is  designed  for  use  by  State  and  county  extension 
workers  who  have  become  interested  in  the  evaluation  process,  or  who  wish  to  review 
major  principles  involved. 


Evaluation  Is  a  Vital  Part  of  Extension  Teaching 

The  Extension  Service,  through  its  many  educational  programs 
and  activities,  is  attempting  to  bring  about  changes  in  what  people 
do,  and  changes  in  their  knowledge,  interests,  understandings, 
attitudes,  and  skills. 

Extension  evaluation  is  the  process  of  determining  the  extent  and 
the  way  these  changes  are  accomplished.  We  need  to  evaluate 
teaching  methods  and  results  in  order  to  complete  any  educational 
work.  This  evaluation  work  will  reveal  "how  you're  doing." 
Then  you'll  have  evidence  of  results  of  your  work. 

When  planned-for  changes  in  people  occur  with  expected  fre- 
quency or  speed,  considering  the  effort  we  put  forth,  we  may  feel 
confidence  in  our  work.  On  the  other  hand,  if  planned-for  changes 
occur  too  infrequently,  or  only  unplanned  changes  occur,  we  all 
realize  that  we  need  to  reorganize  or  modify  our  plans  to  assure 
desired  evidences  of  success. 

We  should  evaluate  small  pieces  of  our  work,  such  as  letters, 
talks,  and  farm  visits,  just  as  carefully  as  our  programs.  In  fact, 
it  is  often  more  practical  to  evaluate  educational  work  "in  pieces" 
than  to  try  to  evaluate  the  results  of  a  total  program  after  it  is 
completed. 

It  is  important  that  we  determine  the  results  of  an  educational 
effort  in  a  reliable  and  objective  manner.  Otherwise,  we  may  draw 
wrong  conclusions.  Such  conclusions  may  give  approval  to  in- 
adequate work  rather  than  suggest  areas  where  improvement  is 
needed.  Unreliable  evaluation  might  even  show  effective  educa- 
tional work  as  being  ineffective. 

We  all  have  six  principal  keys  to  open  the  doors  to  reliable  and 
objective  extension  evaluation. 


Six  Keys  to  Evaluates  Extension  Work 
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Statement  of  Objectives 

mmam     1 .  State  the  objectives  of  an  action  or  activity 
^wBk        to  be  evaluated  in  terms  of  behavior  changes 
IHk     in  the  people  who  are  to  do  the  learning. 

This  is  the  first  key  to  reliable  evaluation.  You  need  to  decide 
the  changes  in  people  that  you  are  to  help  bring  about.  Determin- 
ing whether  or  not  these  changes  occur  is  the  process  of  evaluation. 
Therefore,  it  is  important  that  you  carefully  define  and  clarify  these 
planned-for  changes  at  the  beginning  in  the  statement  of  objectives. 

For  example: 

THIS       w     Farmers  to  learn  which  crop  variety  is  best  adapted 
to  their  soils. 

NOT  THIS      rd^m/ormJfcrmers  about  the  crop  variety  which 

is  DSku($pted  to  their  soils. 
NOR  THIS     TohoM*  meetings  on  crop  variety  suitable  for  farms 

In  the  first  statement,  behavior  change  in  the  learner  is  stressed; 
in  the  other  two,  activity  of  the  Extension  Service  worker  is 
emphasized. 

Starting  from  the  first  statement,  you  can  evaluate  by  determining 
whether  the  farmers  learned  the  desired  specific  information. 
Starting  from  the  second,  you  would  have  to  find  out  only  if  you 
yourself  had  given  out  the  information.  Starting  from  the  third, 
only  the  Extension  worker's  action  concerning  a  teaching  method 
is  included;  the  learners  are  not  mentioned.  Determining  whether 
a  person  held  a  meeting  is  very  incomplete  educational  evaluation. 


Source  of  Evidence 


2.  Only  those  people  whom  you  try  to  reach  can 
provide  proof  of  your  success  or  failure. 


We  learned  in  key  1  that  the  objective  of  your  educational  pro- 
gram or  action  should  be  stated  to  define  clearly  the  desired  behavior 
change.  So  also  should  your  teaching  activities  be  planned  to  reach 
those  people  whom  you  want  to  reach,  and  to  make  learning  most 
possible  for  them. 

Some  kinds  of  learning  can  result  from  a  variety  of  teaching 
methods,  while  others  require  particular  teaching  methods.  For 
example,  a  homemaker  may  learn  the  importance  of  a  balanced  diet 
through  reading,  listening  to  talks,  participating  in  discussions,  and 
other  ways.  We  all  know,  though,  that  a  learner  cannot  learn 
certain  physical  skills  without  actually  doing  them. 

The  teaching  methods  you  use  or  supervise  determine  which  people 
could  have  been  influenced  to  learn.  These  people,  then,  by  their 
performance,  are  the  only  ones  who  can  provide  evidence  of  teaching 
success. 

To  determine  who  might  have  learned  something  from  your  teach- 
ing, you  need  to  know  three  things:  (1)  What  you  can  expect  from 
different  kinds  of  teaching  methods,  (2)  the  possible  chain  reac- 
tion which  your  teaching  of  certain  kinds  of  information  can  set 
up,  and  (3)  whether  or  not  the  learners  have  the  facilities  to  prac- 
tice and  acquire  skill  in  what  you  have  been  teaching. 

Suppose  you  define  incorrectly  the  people  who  can  be  expected 
to  provide  evidence  of  the  degree  of  success  of  your  teaching;  this 
mistake  may  cause  evaluation  to  unduly  emphasize  lack  of  success 
just  as  often  as  it  may  overemphasize  success. 

Objective:  Homemakers  to  acquire  skill  in  cleaning  a  sewing 
machine. 

Teaching  methods  used  at  a  home  demonstration  club  meeting: 


( 1 )  Lecture, 

(2)  Demonstration, 

(3)  Practice  period. 


Those  who  can  provide  proof  of  success  are: 

THESE  IJf  Those  homemakers  attending  the  meeting.  Also 
family  members  or  neighbors  who  could  have 
watched  member  carry  out  practice,  could  have 
asked  questions  and  had  a  sewing  machine  to 
clean. 

NOT  THESE  All  members  of  home  demonstration  club. 
NOR  THESE  A  makers  in  county. 

A  review  of  your  program  and  plan  of  work  will  enable  you  to 
identify  where  success  and  failure  may  have  occurred,  and  those 
people  to  whom  you  can  go  to  obtain  information  about  them.  This 
analysis  need  not  be  a  formal  process,  but  it  should  be  carried  out 
during  the  process  of  evaluation.  Then  you'll  know  what  evidences 
of  success  to  look  for  and  where  to  look  for  them.  Perhaps  you 
can  examine  each  phase,  activity,  or  job  with  these  questions  in 
mind: 

What  is  the  educational  objective? 
What  subject  matter  have  I  taught? 
What  teaching  methods  did  I  use? 
Who  could  have  been  influenced  to  change? 
What  kinds  of  change  could  I  expect? 

Answers  to  these  questions  lead  you  directly  into  the  next  steps 
of  evaluating  Extension  work. 

But  you'll  still  have  done  some  useful  evaluation  if  you  go  no 
further  than  Keys  1  and  2.  By  studying  your  program  and  plan 
of  work  right  at  your  desk,  you  are  analyzing  your  chances  of 
reaching  your  objectives.  This  study,  by  itself,  gives  you  a  par- 
tial answer  to  your  effectiveness — without  going  beyond  this  point. 
Naturally  a  complete  evaluation  using  all  six  keys  is  better  than 
just  the  first  two  alone. 

At  this  stage,  let  us  recognize  that  it  is  not  always  possible  to 
find  out  whether  or  not  people  really  did  learn.  Sometimes  we 
can  only  determine  how  good  the  opportunity  was  for  them  to  learn. 
Do  not  be  satisfied  with  evaluating  only  the  opportunity  to  learn  if 
you  have  any  chance  to  measure  your  success  in  bringing  about 
actual  learning. 


Representative  Sample 


3.  Those  persons  who  actually  provide  the  evidence 
of  success  must  be  representative  of  all  whom 
you  tried  to  reach. 


Extension  workers  usually  reach  more  people  than  they  can  col- 
lect evidence  from  to  provide  proof  of  their  success  or  failure. 
They  also  teach  more  things  than  they  have  time  to  check  on  each 
year.  Therefore,  it  becomes  necessary  to  get  information  from 
only  a  part  of  the  people  about  only  part  of  the  things  taught. 

When  you  get  information  from  only  part  of  the  people,  you 
are  "sampling"  your  "population."  First,  you  describe  your  "pop- 
ulation" through  the  process  described  in  Key  2.  Then,  from  this 
"population,"  you  select  those  from  whom  you  will  collect 
information. 

You  have  two  ways  of  selecting  people  from  whom  to  collect  evi- 
dence so  that  these  people  will  be  representative  of  the  total  group 
you  tried  to  reach: 

(1)  Use  the  total  group  of  people  who  could  have  learned  what 
you  were  teaching  as  the  source  of  evidence.  This  is  usually  too 
large  a  number  to  reach  with  resources  available. 

(2)  Use  a  random  sampling  method.  Give  every  person  in  the 
total  group  an  equal  chance  to  be  selected.    Most  common  ways  are: 

(a)  Choose  every  "nth"  (3d,  7th,  etc.)  name  from  a 
list  of  names  (of  people,  clubs,  and  so  forth).  Be  sure  the  list 
includes  all  names  in  total  group. 

(b)  From  a  map  pick  every  "nth"  geographic  area 
which  contains,  on  an  average,  a  predetermined  average  number  of 
homes. 

Method  (b)  is  not  very  practical  when  people  from 
whom  you're  collecting  evidence  live  in  only  a  part  of  the  homes  in 
the  areas.  It  is  necessary,  however,  when  names  and  addresses  are 
not  available. 

Whether  you  follow  method  (a)  or  (b),  select  a  sample 
large  enough  to  assure  that  most  of  the  characteristics  of  the  total 
group  will  be  represented.    This  minimizes  sampling  error,  and 


allows  for  adequate  analysis  of  the  information.  Statisticians  on 
the  land-grant  college  campuses  can  help  you  decide  the  proper 
size  of  sample.    The  size  depends  on : 

The  facilities  and  time  to  collect  information. 
The  need  for  statistical  accuracy. 

The  uniformity  or  complicated  nature  of  the  total  group. 
Plans  to  analyze  the  information  collected. 

It's  not  enough  to  just  send  or  take  a  questionnaire  or  record  form 
to  the  total  group,  or  to  a  representative  sample  of  the  total  group. 
Filled-out,  or  "answered"  questionnaires  or  record  forms  must  be 
returned  from  all,  or  a  very  high  share  of  those  contacted. 

Total  group  you  were  trying  to  reach:  All  dairy  farmers  in  county. 

Reliable  sample: 

THIS        m    Every  "nth"  name  from  list  of  dairy  farmers.  See 
~     that  each  farmer  selected  answers  the  questionnaire. 
NOT  THIS     Questionnaire  sent  to  all;  use  those  which  are 

NOR  THIS     ColH  {evidence  from  dairy  farmers  who  call  at 
yapronBe  or  whom  you  visit  during  routine  work. 

This  may  sound  like  we're  discouraging  use  of  the  mail  ques- 
tionnaire. Not  at  all.  This  method  is  often  highly  regarded  because 
of  its  convenience.  You  can  use  it  effectively  if  you  send  it  to  a 
representative  sample  of  people,  and  continue  the  followup  until  a 
very  high  proportion  of  returns  comes  in. 

Collecting  evidence  of  success  from  people  who  do  not  represent 
the  larger  group  occurs  very  often  because  (1)  it  is  less  trouble 
to  collect  evidence  from  some  groups  of  the  population  than  from 
others,  or  (2)  the  sample  is  believed  to  be  representative  even 
though  it  is  not. 

Evidence  from  a  nonrepresentative  group  is  no  better  than  no 
evidence  at  all,  and  often  is  so  misleading  that  it  is  worse.  If  it  is 
too  much  trouble  to  get  evidence  from  a  representative  group  in 
order  to  draw  reliable  conclusions  for  the  total  group,  it  is  too  much 
trouble  to  evaluate.  Because  teaching  is  not  complete  without  eval- 
uating, failure  to  evaluate  properly  is  admission  that  complete 
teaching  is  too  much  trouble. 


Appropriate  Methods 


4.  The  methods  of  obtaining  evidence  must  be  ap- 
propriate to  the  kinds  of  information  being 
collected. 


A  fourth  key  to  effective  evaluation:  Remember  that  the  method 
used  to  collect  evidence  depends  on  the  kind  of  evidence,  and  on  the 
resources  you  have  to  collect  it. 

Evidence  of  educational  success  is  that  which  tells  whether  or  not 
learning  has  occurred.  For  each  desired  behavior  change,  you 
must  determine  whether  the  things  you  can  see,  or  hear,  or  feel  will 
tell  you  the  change  has  occurred.  Sometimes  evidence  is  in  the 
form  of  action  carried  on  by  the  learner;  at  other  times  it  might  be 
the  product  resulting  from  some  action;  or  it  may  be  what  the  learner 
passes  on  to  someone  else. 

Things  you  can  see  can  be  recorded  on  observation  sheets.  How- 
ever, you're  seldom  present  to  see  people  carry  out  what  was  taught, 
so  you  have  to  resort  to  forms  filled  out  by  the  learner.  Where  the 
right  situation  exists,  you  can  use  personal  observation  to  obtain 
evidence  of  changes  in  behavior,  such  as  changes  in  knowledge,  at- 
titudes, appreciation,  and  understanding.  Otherwise,  you'll  have  to 
resort  to  questionnaires  or  tests  filled  out  by  the  learner. 

A  great  deal  of  evaluation — of  letters,  of  home  visits,  of  meet- 
ings— is  carried  out  without  recording  the  results.  However,  it  is 
usually  necessary,  in  addition,  to  carry  out  a  systematic  plan  of 
evaluation  which  requires  forms,  or  questionnaires,  or  written  ob- 
servations of  some  kind.  Such  a  plan  should  be  made  out  when  the 
teaching  plan  is  set  up.  If  it  was  not  done  then,  it  is  better  done  later 
than  not  at  all.  Regardless  of  the  method  used,  or  when  it  is 
planned,  keep  in  mind  that  evaluation  is  in  the  terms  of  changes  in 
the  people,  not  in  terms  of  what  you  do. 

Another  important  point  to  consider  in  deciding  how  to  collect  and 
record  evidence  is  determining  how  to  set  up  a  base-mark  from  which 
to  measure.  The  result  of  effective  education  is  a  behavior  change; 
therefore,  it  is  necessary  to  measure  changes,  from  one  time  to 
another. 


Behavior  change  to  evaluate: 


Youth  to  learn  to  be  more  interested  in  his  community. 

THIS        &  Recorded  observation  of  what  he  does  in  his  com- 

^  munity  before  and  after  the  teaching. 

NOT  THIS  Amfcyfki  he  is  more  interested  in  his  community. 

NOR  THIS  As^p^^what  he  has  just  done  in  his  community. 

The  behavior  change  used  in  these  examples  is  one  of  attitude 
change.  Attitude  change  is  indicated  by  what  a  person  does,  and 
not  by  what  he  says.  Therefore,  the  first  method  is  much  more  valid 
than  the  second  method.  The  third  method  does  not  indicate  any 
change  in  attitude.  The  person  may  have  done  these  same  things 
for  many  years,  both  before  and  after  the  teaching,  which  would 
mean  no  change  in  his  attitude. 

These  examples  illustrate  only  the  one  point:  The  need  to  deter- 
mine whether  or  not  a  behavior  change  occurred.  Another  point  to 
keep  in  mind  is  the  need  to  record  exact  methods,  amounts,  lengths 
of  time,  and  so  forth  at  the  time  the  action  takes  place.  When  you 
need  that  kind  of  data  as  evidence,  do  not  depend  on  memory;  you 
should  set  up  a  system  of  periodic  recording. 

Here's  a  third  point  of  importance,  when  deciding  on  methods 
of  collecting  information:  Who  collects  it  as  well  as  who  provides 
it.  Often  children  cannot  provide  evidence  of  their  learning,  and, 
in  some  of  the  more  difficult  areas,  adult  learners  cannot  provide 
evidence.  In  many  cases,  when  adults  would  provide  some  people 
with  reliable  evidence,  they  would  not  others.  Always  ask  yourself: 
Who  can  provide  reliable,  useful  information?  Who  can  get  it 
from  them,  intelligently  and  with  good  public  relations? 

Often  an  important  decision  for  Extension  workers  is  whether 
to  use  a  mail  questionnaire,  a  personal  interview,  or  a  group 
questionnaire. 

The  mail  questionnaire,  although  inexpensive,  is  often  not  re- 
turned; it  has  to  be  short  and  simple. 

The  personal  interview,  with  or  without  a  questionnaire,  is  ex- 
pensive but  can  obtain  the  most  reliable  information. 

The  group  questionnaire  is  less  expensive  but  the  group  members 
often  want  to  all  give  the  same  answers.  With  school  children,  it 
is  the  most  appropriate  method  for  most  subjects. 


Reliable  Questions 


5.  Word  questions  carefully  so  you  obtain  reliable, 
unbiased  data. 


You  can  ask  questions  directly  or  indirectly,  or  use  them  as  guides 
for  observation.    A  question  is  reliable  if  you  ask  it  in  such  a  way, 
and  under  such  conditions,  that  you  can  have  faith  in  the  answer. 
Here  are  a  few  of  the  "Do's"  for  those  who  construct  questions: 
Ask  questions  which  are  clear,  concrete,  and  have  definite 
answers. 

Ask  questions  that  are  easy  to  understand. 
Ask  questions  without  a  clue  to  the  answer  desired. 
Avoid  questions  which  involve  more  than  one  idea. 
Provide  adequate  space  to  write  each  answer.  Whenever 

possible  use  a  check  for  an  answer. 
Field-test  (pretest)  all  questions. 

See  that  the  time  period  covered  by  each  question  is  clear. 
For  those  who  ask  the  questions,  ask  every  question  exactly  the 
same  every  time.    Do  not  imply  answers  when  explaining. 
When  asking  questions  about  a  tour: 

THIS      »       Did  you  see  any  new  methods  of  pasture  treatment 

on  the  tour?   Yes   No  

It  YES,  what  were  they?   

(Allow  space  to  write.) 

\OT  THIS      W  mt  new  mjflbds  of  pasture  treatment  did  you  see 


(Allow  space  to  write.) 

j  think  the  tour  was  helpful?  Yes  

't  know  

The  first  part  of  the  first  question  is  specific.  The  second  part 
of  the  question  is  a  check  on  the  reliability  of  the  response  to  the 
first  part. 

The  second  question  starts  out  by  assuming  everyone  saw  some- 
thing new.    This  forces  an  answer. 

In  the  third  question  the  correct  answer  is  implied,  and  the 
meaning  of  the  word  "helpful"  is  obscure. 


Plan  To  Use  Results 

^(BL      6.  Decide  how  you  will  analyze  and  use  your 
SV(\         evaluation  results  before  evaluation  is  done. 

\ 

We  don't  collect  evidence  just  to  be  doing  something.  Evaluation 
includes  studying  the  results  and  applying  the  analysis  of  them  to 
objectives  and  methods  to  see  where  changes  need  to  be  made. 

Because  evidence  of  educational  activity  is  invariably  collected 
from  a  number  of  people,  you  as  evaluator  first  add  up  the  evidence 
you  have  collected. 

Converting  the  totals  into  percentages  or  averages  provides 
statistics  that  speak  for  people  as  groups  rather  than  as  individuals 
and  enables  comparison  of  groups  of  different  sizes.  If  you've  done 
your  sampling  properly,  you  can  apply  percentages  and  averages  to 
the  total  group  from  which  the  respondents  were  drawn. 

In  this  final  step  of  summarizing  and  analyzing  data,  your  inter- 
pretation of  them  is  of  greatest  importance. 

It  may  interest  you  or  others  that  in  a  given  area  10  percent  of 
the  dairy  farmers  are  organized  in  an  artificial  insemination  organi- 
zation or  that  50  percent  of  the  farm  homemakers  with  less  than 
high-school  education  have  asked  for  discussions  of  rural  health 
facilities.  However,  study  these  percentages  with  questions  like 
these  in  mind: 

Is  this  percentage  high,  low,  expected,  or  unexpected? 
What  have  I  done  or  not  done  to  make  it  high,  or  keep 

it  so  low? 
What  other  factors  are  related  to  it? 
How  should  I  change  my  methods  or  program  to  bring 

about  a  different  kind  or  a  different  amount  of 

change? 

When  statistical  data  and  other  information  collected  during 
evaluation  have  been  summarized,  analyzed,  and  used,  two  results 
occur:  (1)  The  program  fits  the  needs  and  (2)  You  are  more  effi- 
cient as  an  Extension  Service  worker. 
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WHEN  WE  EVALUATE 
EXTENSION  WORK— 

•  We  do  so  in  terms  of  objectives. 

•  We  base  it  on  evidence  from  people 

we  try  to  reach. 

•  We  record  the  evidence. 

•  We  use  the  findings. 


Cooperative  Extension  Work:  United  States  Departmeat  of  Agriculture 
and  State  Land-Gram  Colleges  and  Universities  Cooperating. 


